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ABSTRACT 

On the basis of the Research Project funded by the Spanish 
Ministry of Education under the title "Design, production and 
evaluation of a 2.0 learning environment for faculty training in the 
use of Information and Communication Technologies (ICTs)” 
(EDU2009-08 893), experts have used the external competence 
coefficient to evaluate the different dimensions of Personal 
Learning Environments (PLE), namely: technical and aesthetic 
aspects, ease of navigation, or quality of the didactic elements 
that make up the environment. A quantitative methodology along 
with a questionnaire prepared by the author served this purpose. 
The results obtained highlight technical environment operation, 
the tools forming the PLE, or the learning object repository as 
being “very positive.” In conclusion, experts emphasise the user- 
friendliness of environment and tools alike, as well as the 
educational aspects of the contents available in materials guides 

KEYWORDS: PERSONAL LEARNING ENVIRONMENT, 

TEACHER TRAINING, HIGHER EDUCATION, INTERNET 
RESOURCES, INSTRUCTIONAL TECHNOLOGY 


1 INTRODUCTION 

University teacher training stands out as one of the most 
commonly discussed topics in studies or papers about the 
incorporation of Information and Communication Technologies 
(ICTs) into higher education (Cabero, 2006; Bozu & Canto, 
2009; Bullon et al., 2009; Mehdinezhad, 2012; Reilly, 
Vandenhouten, Gallagher, & Ralston, 2012; Terantino & 
Agbehonoou, 2012; Vazquez, Alducin, Marin, & Cabero, 2012). 
There is no doubt whatsoever at this stage that the degree of 
success or failure of some models and tools used in teaching¬ 
learning processes largely depends on the good, bad or non¬ 
existent training of teachers for the correct technical and didactic 
utilisation of the aforesaid models and tools (Imbemon, 2008). 

The degree of peculiarity associated with the social web or 
Web 2.0 tools, which characteristically are born, grow and dis¬ 
appear at great speed (Bennet, Bishop, Dalgarno, Waycott, & 
Kennedy, 2009; Castano, 2009; Roig & Laneve, 2011; Navas, 
2012) could be added to all of the above. On certain occasions. 
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teachers might actually not have enough time to be trained on 
how to use them or even to become familiar with them. 

All of this implies a formal change for teachers both in terms 
of mind-set (Cabero, Marin, & Infante, 2011) and with regard to 
action design, planning and implementation, which in turn forces 
us to stop pondering so much about how things are taught and to 
place more emphasis on how people learn, though never losing 
sight of the link that connects both areas. In short, it involves the 
development of teacher competences such as those described by 
Valcarcel (2003) in the following terms: 

— Cognitive competences specific to teacher duties in a 
particular discipline. They involve appropriate training -i.e. 
a broad knowledge of the specific discipline and 
pedagogical field that can permit to develop relevant 
training activities to support student learning. 

— Meta-cognitive competences that allow teachers to become 
reflective professionals who are self-critical with their 
teaching for the purpose of revising it and ultimately 
improving it in a systematic way. 

— Communicative competences, closely related to the proper 
use of scientific languages (numerical, alphabetical, 
graphics, etc.) and their different registers (articles, reports, 
essays, lectures, lessons, etc.). 

— Managerial competences related to the efficient 
management of teaching and teaching resources in different 
situations and learning environments. 

— Social competences connected to actions of leadership, 
cooperation, persuasion, teamwork, etc., thus promoting 
training and willingness of their students in this area, as 
well as their own professional development, primarily 
within the European Higher Education Area. 

— Emotional competences ensuring attitudes, motivations and 
behaviours that contribute to a responsible way of teaching 
committed to the achievement of the desired learning 
objectives. 

Society is currently immersed in what has come to be known 
as the social web (O’Reilly, 2005; Downes, 2007; Selwyn & 
Gouseti, 2009; Brown, 2010; Castaneda, 2010), the most 
defining traits of which are summarised and defined below 
(Castano, Maiz, Palaci, & Villaroel, 2008): 

a) The web as a platform: dynamic applications, 
collaborative applications, and simple, intuitive tools. 

b) Programming and composition become easier: AJAx 
technology, transition towards XML, separation between 
design and content; ease of interoperability, standards 
and software not limited to a single device. 
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c) Software as a service: online services and applications 
from the browser, interoperability between services and 
applications, standards. 

d) Making the most of collective intelligence. 

e) Everyone is an author who can publish: reading and 
writing networks, simple and powerful tools (blogs, 
wikis, photos, videos, podcasts, etc.) 

f) Content management: creation and sharing of 
knowledge, micro-contents, using metadata, syndication, 
as well as tagging and folksonomy. 

All in all, this more socially connected Web allows people to 
contribute as much as they can consume (Anderson, 2007). Most 
of the tools and resources available (wikis, blogs, YouTube, 
social networks, bookmarking, etc.) focus on enabling and 
promoting user-generated content that can be later distributed 
through the participation, interaction and collaboration of 
everyone -hence its “social” label. 

The most significant aspect from an educational standpoint is 
perhaps that this change involves not only a variation in the tools 
or resources incorporated: it must go further and promote a 
change of attitude where students have the chance to assume an 
active role in the teaching and learning process of which they are 
an essential part by means of self-regulated learning. In turn, the 
teacher owns the different skills and abilities needed to modify 
the design and planning of instructional situations in the context 
of teaching. 

Several principles can be associated with everything related to 
the social web (Palomo, Ruiz Palmero, & Sanchez Rodriguez, 
2008): a) An attitude against technology: Web 2.0 has 
consistently had a strong impact on the field of knowledge; b) 
The permanent Beta: the tools and resources are continuously 
developed, new features are often added, distributing the 
software as a service rather than a product; c) The right to mix a 
number of reserved rights: Creative Commons (Kapitzke, 
Dezuanni, & Iyer, 2012) devised a new licensing system for 
content distribution and use that allows users to share their 
knowledge without violating privacy rights; d) Emergent: free 
software which contains mechanisms favouring intrinsic patterns 
and structures in interactions between individuals; f) hackbility: 
or the ability to experiment with various data sources, mixing 
them to create a new product. 

It is within this specific context that the concept of Personal 
Learning Environments (PLEs) appeared some years ago as a 
new way to understand how students learn as well as how educa¬ 
tors teach (Attwell, 2007; Schaffert & Hilzwnsauer, 2008; Adell 
& Castaneda, 2010; Santamaria, 2010; Modritscher et al., 2011; 
Barroso, Cabero, & Vazquez, 2012; Cabero, 2012). 

PLEs appeared as “a new construct in the e-learning literature 
which finds its support on social media and steadily gains 
ground in the e-learning field as an effective platform for student 
learning” (Dabbagh & Kitsantas, 2012). 

It could be understood as a possibility to incorporate the tools 
described above as “social” or “2.0” to the instructional process 
from a new perspective that took as its reference the student’s 
acquisition of new ways to work in the virtual learning context. 

At the same time, and from a first perspective, Barroso et al. 
(2012) include authors such as Fiedler & Pata (2009), Amine 
(2009), Reig (2009) or Henri, Charlier & Limpens (2008) as the 
ones who consider PLEs a self-defined collection of resources, 
services, tools and devices which can help teachers and students 
shape their personal learning and knowledge networks. 

From this perspective, they are presented as an option, 
sometimes in contrast and sometimes as an added option, to the 


traditional learning platforms. Nevertheless, from a different 
perspective, the same authors (Barroso et al., 2012) highlight 
that the emphasis lies on the educational application component 
and its consideration as a new educational methodology: “we 
could say that the concept of PLE includes integrating elements 
of both formal and informal training in a unique learning 
experience, as well as the use of social networks that can cross 
institutional boundaries and the use of network protocols. Here 
the focus is on the students and on the decisions they make to 
customise and self-regulate their learning.” 

It is from this idiosyncrasy that the need arises to design and 
implement a study combining these two concepts or themes: 
teacher training and PLE. 

This was the challenge assumed in the research project 
“Design, production and evaluation of a learning environment 
for faculty training in the educational use of Information and 
Communication Technologies” (DIPRO 2.0), funded by the 
Spanish Ministry of Education (EDU2009-08893EDUC), and 
directed by Dr. Julio Cabero (University of Seville), of which 
some outstanding results have already been presented in 
different publications (Cabero et al., 2011; Cabero & Marin, 
2012 ). 

2 OBJECTIVES 

Even though the development of this research revolved around 
four basic objectives that interested readers can find in other 
published works (Cabero & Marin, 2012), the present article has 
as its main aim: 

a) To validate the telematic learning environment, regarding 
both the contents proposal and the different communication 
tools (blogs, wikis, etc.) created. 

The first step taken to achieve that aim was the construction of 
a personal learning environment that can be found in the 
following address: http://tecnologiaedu.us.es/portal/ . to which 
another environment had to be added for the management of 
materials and contents, also known as learning objects: 
http://tecnologiaedu.us.es/dipro2 . 

3 MATERIAL AND METHODS 

An instrument was developed within the project to carry out the 
PLE evaluation process; it could collect information about the 
two environments developed, both independently and jointly. 
The instrument simultaneously incorporated questions aimed to 
obtain some background information about the experts, such as: 
qualification, institution where they worked, specific job, etc. 
The total number of items in the instrument was 38, of which 29 
were intended to collect information from the environment. The 
remaining nine focused on obtaining information about the most 
relevant features of expert evaluators. 

A number of previous instruments used in other research 
studies were taken into account for the construction of the 
instrument used in this study (Cabero, 2006; Vazquez et al., 
2012 ). 

This strategy based on using experts in educational research 
for the evaluation of teaching materials is quite usual (Salinas, 
2004; Barroso & Cabero, 2010) since it has three advantages: 
the theoretical quality of the responses achieved; the level of 
depth in those responses; and the possibility to obtain detailed 
information. 

As for the ways in which the strategy can be implemented, 
several options exist: 
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a) Individual aggregation of experts (it implies obtaining 
the information individually from different experts 
without them being in contact); 

h) Delphi method (which individually and anonymously 
collects the experts’ opinion and gives back to them the 
collective proposal for their review and agreement) 
(Malla & Zabala, 1978; Romero, 2012); 

c) Nominal group technique (experts contribute with their 
information individually, after which an agreement is 
reached in a group meeting); 

d) Consensus method (the selected experts reach an 
agreement as a group and all together). 

The first option was adopted for our study; that is, selecting 
and aggregating each expert individually. Different criteria may 
guide this process (Brill, Bishop, & Walker, 2006; Garcia & 
Fernandez, 2008) hut these were the ones followed in the present 
study: 

a) Work experience in the field of Educational Technology, 
Information Technology and Communication Applied to 
Education, or e-leaming 

h) Experience in virtual training, e-leaming and PEE 

c) Training experience of over five years 

d) Belonging to different Spanish and Latin American 
Universities 

e) Having previously collaborated on other research works 

A total of 84 experts were invited to participate in the 
evaluation, 70 of whom responded in time for the data 
collection. The “expert competence coefficient” or K coefficient 
(Garcia & Fernandez, 2008; Lopez, 2008; Blasco, Lopez, & 
Mengual, 2010; Mengual, 2011) was applied on them in order to 
narrow our selection. This coefficient has already been 
incorporated in numerous works: Cuesta and Godinez, (2008); 
Lopez, Stuart, and Granado (2011, 2012); Gongora, Hernandez, 
Garcia, and Sanchez (2009); Gonzalez and Vidaud (2009); 
Betancourt, Cobelo, and Zabala (2010); Herrera, Iglesias, 
Contreras, Lopez, and Sanchez (2010); Mengual (2011); y Zayas 
( 2011 ). 

The coefficient is obtained by applying this formula: 

K='A(Kc + Ka) 

where Kc is the “coefficient of knowledge” or information that 
the expert has about the topic or problem. Its calculation is based 
on the evaluation made by the expert on a O-to-10 scale, 
multiplying it by 0.1; in turn, Ka is the “coefficient of 
argumentation” or the foundation of the expert criteria, and it 
derives from assigning a number of ratings to different 
argumentation sources that the expert might have used. 

The values obtained after the application of the formula 
permitted to establish that experts with values below 0.8 should 
not be used for the study, as a result of which 17 of them were 
removed from the group, leaving a total number of 57 experts. 

Regarding the academic titles of the experts that finally took 
part in the study, 47 (82.5%) were doctors, 8 (14.0%) had a 
master’s degree, 1 (2.9%) had a bachelor’s degree, and 1 did not 
fill in this item. 

Most of them (/=56, 98.2%) developed their professional 
activities in universities, and only one of them (1.8%) did not. 
The vast majority were teachers (/=51, 89.1%), followed by 
those who carried out management as well as teaching tasks 


(/=4, 7.0%); only 2 of them exclusively carried out management 
activities (/=2, 3.5%). As for whether they had taught any 
subjects related to ET and ICTs, most of them claimed that they 
had (/=55, 96.5%), and also that they had published or taken part 
in a publication related to this research topic (/=54, 94.7%). 

4 RESULTS 

First of all. Table 1 shows the mean values and standard 
deviations achieved in the three main dimensions that 
constituted the data collection instrument. For a correct 
interpretation of them, note that the answering options were: 
“MP (Muy positivo) = Very positive / I strongly agree (6)” to 
“MN {Muy negativo) = Very negative /1 strongly disagree (1)”, 
and all of them had six response options. 


Table 1. Mean values and standard deviations carried out by experts in 
the environments perceived all together and separately 


Dimensions 

M 

SD 

(1) Assessment of both values jointly 

5.21 

.62 

(2) Quality of environment (a) to create a PLE 

5.16 

1.05 

(3) Quality of environment (b) to create a PLE 

5.07 

.67 


The mean values achieved show that the experts gave the 
environments a positive assessment both jointly and 
individually. Moreover, the low standard deviations indicate that 
there was a high degree of similarity between the answers. 

The scale used to rate the two modules together collected data 
about three sub-dimensions at the same time: technical and 
aesthetic aspects; ease of navigation and scrolling through the 
environment; and program’s tutorial/guide. The results obtained 
can be seen in Table 2. 


Table 2. Mean values and standard deviations carried out by experts 
within the environments in three sub-dimensions 


Joint assessment of the environments 

M 

SD 

(1) Technical and aesthetic aspects, ease of navigation 

5.19 

.9 

(2) Scrolling through the environment 

5.18 

.81 

(3) Program tutorial/guide 

5.38 

.82 


The results offered below show the values achieved in each of 
the items for the instrument developed, grouped by different 
dimensions, where the technical and aesthetic aspects formed 
part of the first dimension. Table 3 provides the results obtained. 


Table 3. Expert evaluation of technical and aesthetic aspects in both 
environments 


Technical and aesthetic aspects 

M 

SD 

(1.1) Correct operation of the links between the different 

5.21 

.94 

parts of the environment (there are broken links): 

(1.2) The letters’ size and font type is: 

5.02 

1.03 

(1.3) The size of the graphics, texts, animations,... is: 

5 

1.02 

(1.4) The text page length is: 

5.14 

.92 

(1.5) In general, the aesthetic of the environment could be 
considered: 

5.05 

1.01 

(1.6) In general, the technical program operation is: 

5.44 

.68 

(1.7) In general, the time to access the different program 
parts is: 

5.39 

.73 

(1.8) In general, the information display on the screen is: 

5.19 

.88 
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Expert evaluation shows that the values obtained were slightly 
above positive regarding the following elements: correct 
operation of the links between the different parts of the 
environment, letters’ size and font type, graphic sizes, texts, 
animations, text page length, environment’s aesthetic design, 
technical program operation, time to access the different 
program parts, and information display on the screen. Mean 
scores below five (“positive”) were observed in none of the 
items, the evaluation ("In general, I would rate technical 
program operation as:") being close to the option “very positive” 
(5.44) in some of them. 

As for the dimension ease of navigation and scrolling through 
the environment, the values obtained (Table 4) were also 
situated above 5. More specifically, this indicated that their 
appreciation of the following areas was positive: ease of use and 
management of the environment for the user, user’s 
understanding of technical environment operation, 
recognisability of the overall design in the web environment 
developed, environment accessibility/usability, and environment 
flexibility. 


Table 4. Expert evaluation on ease of navigation and scrolling through 
both environments 


Ease of navigation and serolling throngh the 
environment 

M 

SD 

(2.1) In general, the ease of use and management of 

5.37 

.1 

the environment for the user is: 

(2.2) In general, user’s understanding of technical 
operation of the environment is: 

(2.3) From your point of view, the recognisability of 

5.19 

.77 

the overall design in the web environment 
developed is: 

5.16 

.77 

(2.4) From your point of view, the environment 

5.16 

9 

accessibility/usability is: 


(2.5) In your opinion, the environment flexibility is: 

5.02 

.92 


The environment included a support “guide/tutorial” and the 
experts evaluated it as well. In this sense, the values obtained 
were highly positive, both regarding the ease to understand it 
and its simplicity (Table 5). 


Table 5. Expert evaluation of the program’s “Guide/Tutorial” 


Program guide / tutorial 

M 

SD 

(3.1) In general, the tutorial's usefulness and 



understandability when trying to know how the 
environment works is: 

5.32 

.91 

(3.2) The tutorial is simple and understandable 

5.37 

.77 

(3.3) How would you rate the tutorial's ease of use? 

5.42 

.78 


Below are the results obtained for the environment defined as 
"Personal learning environment (a)." As shown in Table 6, the 
data was highly positive, especially regarding the following 
items: 

— From your point of view, the environment developed 
features the most common tools used to create a personal 
learning environment (blog, social networks, wikis, ...) 
(5.29), and 

— Rate from an educational point of view the quality of the 
different tools that are incorporated into the environment 
(5.23). 


Table 6. Evaluation of the environment developed 


Quality of the environment (a) to create a PLE 

M 

SD 

(4.1) From your point of view, the environment developed 
includes the most common tool to create a PLE (blogs, 

5.29 

.97 

social networks, wikis, etc.) 

(4.2) How would you rate the integration in the 

environment of the different tools (blog, wiki, LMS, 

5.16 

.16 

etc.) which help to create a PLE? 

(4.3) How would you rate the usefulness of the tools used 
to build a PLE? 

5.17 

.06 

(4.4) Rate, from an educational point of view, the quality of 
the different tools that have been incorporated to the 

5.23 

.98 

environment 

(4.5) How flexible is the environment? 

5.03 

1.17 

(4.6) Do you think the environment can be useful for the 
educational practice and help the students to build 

5.1 

1.11 

their own PLE? 




Finally, Table 7 presents the values obtained regarding the 
additional environment which could be considered as a “learning 
object repository.” Its assessment was also positive, the highest 
scores corresponding to the following items: 

— It is easy to understand how the environment works (5.21). 

— Enough materials or learning objects are offered for each 
unit (5.19). 

— Rate from an educational point of view the quality of the 
different tools that are incorporated in the environment 
(5.11). 

In keeping with these results, it must be recognised that the 
scores obtained were significant in the rest of the items too: The 
activities which are presented in each unit suffice for the 
acquisition of the skills that it establishes (4.94); The structure in 
which the “materials guide” has been developed can be 
considered as (4.91); The structure in which the “activity guide” 
has been developed is, in relation to the objectives that need to 
be reached: “(4.91); and The procedure to transfer the different 
learning objects is quite easy (4.90). 


Table 7. Evaluation of the environment developed 


Quality of the environment (b) to create a PLE 

M 

SD 

(5.1) It is easy to understand how the environment works 

5.21 

.98 

(5.2) Enough materials or learning objects are offered for 
each unit 

(5.3) The activities which are presented in each unit 

5.19 

.94 

suffice for the acquisition of the skills that it 
establishes 

4.94 

1.03 

(5.4) Assess, from an educational point of view, the 



quality of the different tools that have been 
incorporated to the environment 

5.11 

.95 

(5.5) The structure in which the “materials guide” has 
been developed can be considered as: 

(5.6) The structure in which the “activity guide” has been 

4.91 

.05 



developed is, in relation to the objectives that need to 
be reached: 

4.94 

1.05 

(5.7) The procedure to transfer the different learning 
objects is quite easy 

4.9 

1,14 
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5 CONCLUSIONS 

Even though this stands out as one of the reference topics in 
education at a higher education level, both nationally and 
internationally, it is difficult to find studies and research that can 
help build a theoretical/conceptual framework around it, or 
about the practical part of PLE development and incorporation 
into teaching. Therefore, some of the results already obtained in 
the investigation and previously mentioned in this article must 
be emphasized as being relevant. 

It should be noted, in conclusion, that the significance of the 
findings revolves around two main issues: first, the creation of 
the two environments personally designed and built for the 
development of a PLE in order to guide university teachers’ 
training in ICTs. This becomes even more significant 
considering that a rigorous procedure has been followed 
throughout the present study to select experts evaluating the “a” 
and “b” environments, and in all the different stages that make 
up the project itself: 1. Virtual learning environment design, 
production and evaluation; 2. Pilot study; and 3. Virtual 
environment presentation. 

References have already been made to the progressive 
abandonment of emphasis on telematic learning platforms 
towards more flexible and customisable models (from server- 
based to distributed environments and laptops). A remarkable 
application of this approach appears in Durall, Gros, Maina, 
Johnson, and Adams (2012, 2011) who used the project itself as 
an example of a PLE in practice through the Horizon Project. 
They actually conceive this perspective as an application of tools 
from the field of education within a scope of two to three years: 
"Project DIPRO 2.0 provides university teachers with different 
online environments to access learning objects, guidelines and 
criteria for the evaluation of activities: 
http://tecnologiaedu.us.es/portal/". 

As for the results of expert evaluation, and regardless of the 
positive reviews found both for the technical and the aesthetic 
aspects, or those referring to ease of navigation and the proper 
operation of all the environment parts, one particular aspect 
acquires relevance from a more technical perspective: the 
suitability of combining OKI (Open Knowledge Initiative) and 
Moodle, since this has been configured as a combination of tools 
which can be easily mixed to construct a PLE, hence 
contributing to its use in formal learning contexts. 

At the same time, the design model consisting of materials 
guides, organised as learning objects in different formats, may 
be of interest for university teachers interested in the 
development of materials. With a clear focus on e-activities, it 
becomes significant for the acquisition of skills needed to train 
teachers /students, and emphasises the constructivist approach to 
the learning process as well as the topics that make up each of 
the guides, to which must be added that it meets another of the 
research objectives ("To develop basic themes in consensus 
between different ET professionals on the most significant areas 
in which university teachers should be trained for the didactic 
use of ICTs"). 

One of the big challenges for future research into PLEs from 
an educational standpoint is to unify them with other types of 
more institutional environments (Barroso et ah, 2012), not from 
a technological/instrumental perspective as with the Learning 
Management System (LMS) but rather from a cultural and 
attitudinal reference perspective. 

It is there that most of the problems arise, often caused by the 
university’s inclination towards control and its fear of open. 


flexible and personal environments: “in order for this learning 
approach to be significant, i.e., for it to offer real possibilities to 
acquire high-level institutional learning, two issues possibly 
need to be reviewed: the competences that students have to 
succeed in environments with instructive purposes, and the 
methodological skills owned by the teacher for the incorporation 
of these new approaches into the educational practice.” This last 
aspect re-establishes the link between the need for technical and 
didactic training of university teachers, as pointed out at the 
beginning of this article, but focusing on new tools and, 
therefore, on new teaching and learning approaches. 

Finally, it is worth highlighting that the proposal conducted 
annually by the Centre for Learning and Performance 
Technologies becomes not only necessary but also advisable 
when selecting the tools to configure a PLE, regardless of the 
degree of customisation that will later be reached with each one 
of them. Furthermore, as it happened with the pioneering studies 
on telematic learning platforms, there is a need to focus the 
challenge of educators towards PLE adoption and integration 
from a pedagogical point of view rather than from a technical 
standpoint. 
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